gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
3323 commits 28 branches 278 tags 135 MiB
Commit graph

244 commits

Author SHA1 Message Date
Chris Lu 0fe286a6ca filer: optimization for leveldb, add leveldb2 
large filer db may see high CPU and disk usage due to background compaction
 2019-05-17 17:33:49 -07:00
Chris Lu bf9d490538 fix for tests 2019-05-17 02:28:20 -07:00
Chris Lu 82b0759493 filer: migrating filer store from persisting shorter structured file id instead of a string 2019-05-17 02:03:23 -07:00
Chris Lu f0f981e7c8 webdav: fix directory renaming 2019-05-03 14:12:51 -07:00
Chris Lu bfbecd7253 webdav: fix directory creation 2019-05-03 13:13:11 -07:00
Chris Lu 7064b2ea48 webdav: can write now 2019-05-03 00:55:52 -07:00
Chris Lu b30c14b631 webdav: can read now 2019-05-03 00:24:35 -07:00
Chris Lu 108d0fb08d adjust log level 2019-04-26 09:32:37 -07:00
Chris Lu 07091c44cd lock btree writes 
fix https://github.com/chrislusf/seaweedfs/issues/930
 2019-04-16 19:48:07 -07:00
Chris Lu 4393b99332 add notes 2019-04-16 13:57:45 -07:00
Chris Lu 338e6d60a5 refactor: prepare for snapshotting 2019-04-16 09:55:37 -07:00
Chris Lu b3b42bc947 replicate need to include new entry path 2019-04-16 00:44:31 -07:00
Chris Lu 35aba35c07 Revert "weed volume: disable block cache on leveldb" 
This reverts commit a55e708d36.
 2019-04-09 00:22:40 -07:00
Chris Lu a55e708d36 weed volume: disable block cache on leveldb 
saving 8MB for each volume, but at the cost of slower (offset,size) lookup
 2019-04-09 00:18:32 -07:00
Chris Lu beaa2bd71a weed filer: properly delete cached directory entry 
fix https://github.com/chrislusf/seaweedfs/issues/910
 2019-04-01 17:03:04 -07:00
Chris Lu 2a52e70a41 weed filer: increase default mysql/postgres dir or name length to 65535 2019-04-01 12:37:54 -07:00
Chris Lu b4c276263f weed filer: mysql/postgres use ReadCommitted transaction isolation level for atomic rename 2019-04-01 11:03:04 -07:00
Chris Lu 78ac2bef3c go fmt 2019-03-30 23:09:16 -07:00
Chris Lu 97406333a5 support atomic renaming for mysql/postgres filer store 2019-03-30 23:08:29 -07:00
Chris Lu b92122b885 fix test 2019-03-16 09:50:57 -07:00
Chris Lu 36b632ebcb fix tests 2019-03-16 09:50:21 -07:00
Chris Lu 55bab1b456 add context.Context 2019-03-15 17:20:24 -07:00
Chris Lu cece860bfd add context to all filer APIs 2019-03-15 15:55:34 -07:00
Chris Lu 55c85f3a66 adding context 2019-03-15 15:26:09 -07:00
Chris Lu d312c55bbe file path supports windows, avoiding back slashes 
fix https://github.com/chrislusf/seaweedfs/issues/868
 2019-03-04 13:00:08 -08:00
Chris Lu 2442d56671 adjust package name 2019-02-18 18:03:16 -08:00
Chris Lu 77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
bingoohuang ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
Chris Lu 2a75a36b27 set cassandra ttl with entry.TTLSec 2019-01-08 20:26:19 -08:00
byungjo.yoon 58fd2e5a19 Set redis key ttl with entry.TTLSec 2019-01-09 11:48:19 +09:00
Chris Lu be9a7592a1 incrementally calculate visible intervals 2019-01-05 22:23:44 -08:00
Chris Lu 832ac8129a reduce pointer usage 2018-12-31 15:10:14 -08:00
Chris Lu 0388d421d2 caching visible intervals for read 
speeds up 4x in single thread mode
speeds up 30% in 32 threads mode
 2018-12-30 00:51:44 -08:00
Chris Lu 253f190f48 weed mount add symlink support 2018-12-25 22:45:44 -08:00
Chris Lu 3ac54792e1 paginate when filer deleting and FUSE mount renaming 2018-12-16 23:20:08 -08:00
Chris Lu 1bfb96f34d optimization for reading whole chunk with gzip encoding 2018-12-07 01:57:55 -08:00
Chris Lu c28e8a2397 refactoring 2018-12-06 00:44:41 -08:00
Chris Lu 63619f8b8e fail early if two filer stores are defined 2018-12-06 00:32:36 -08:00
Chris Lu 52b24a9902 default "weed server -filer" to same directory as -mdir 2018-12-05 23:24:25 -08:00
Chris Lu b0838db5a8 support hdfs user name and group names 2018-12-04 00:13:40 -08:00
Chris Lu c6b646e92c skip creation for root 2018-12-02 23:16:17 -08:00
Chris Lu f0677c5af1 default root to 0755 2018-12-02 19:59:47 -08:00
Chris Lu db22666a30 add default root directory 2018-12-02 19:42:50 -08:00
Chris Lu 7a6f49cd42 refactor a bit more 
fix https://github.com/chrislusf/seaweedfs/issues/777
 2018-11-29 00:07:54 -08:00
Chris Lu ceca078acb avoid overwriting file or directory 
fix https://github.com/chrislusf/seaweedfs/issues/777
 2018-11-29 00:00:56 -08:00
Chris Lu 86dd933596 go fmt 2018-11-23 00:26:15 -08:00
Chris Lu 99256100a0 fix tests 2018-11-21 16:35:27 -08:00
Chris Lu e5ad2223a1 add missing intervals 
fix https://github.com/chrislusf/seaweedfs/issues/774
 2018-11-21 16:25:13 -08:00
Chris Lu b282e34dc2 async file chunk deletion 2018-11-20 20:56:28 -08:00
Chris Lu b318167cd3 batch delete file ids 2018-11-20 11:35:08 -08:00
Chris Lu ee3fe07acf fix sort bug 2018-11-18 21:59:53 -08:00
Chris Lu 469863f7f0 merge interval now is quite faster with reusing slices 2018-11-18 21:24:58 -08:00
Chris Lu d2924c7748 merge intervals is a bit faster 2018-11-18 20:31:39 -08:00
Chris Lu cb5a10c6a3 adjust the tests 2018-11-18 17:05:36 -08:00
Chris Lu f1db22d48b add benchmark test for merging intervals 2018-11-18 16:49:08 -08:00
Chris Lu 4fcfc9410f cleanup 2018-11-18 11:51:38 -08:00
Chris Lu 9655dc9ca9 simpler logic 2018-11-18 10:07:30 -08:00
Chris Lu a64613172d bootstrap filer replication with weed filer.export -targetStore=notification 2018-11-03 12:43:45 -07:00
Chris Lu 4c97ff3717 support AWS SQS as file change notification message queue 2018-10-31 01:11:19 -07:00
Chris Lu ff66269b62 use grpc to replace http APIs for batch volume id lookup and batch delete 
1. remove batch volume id lookup http API /vol/lookup
2. remove batch delete http API /delete
 2018-10-14 00:12:28 -07:00
Chris Lu 56a5d5af8d ignore duplicated key error if the store has duplicated primary key prevention 
fix https://github.com/chrislusf/seaweedfs/issues/741
 2018-09-30 00:49:52 -07:00
Chris Lu 7bb62b9f21 delay file creation without chunks from dir.Create to file.Flush 2018-09-22 00:11:46 -07:00
Chris Lu db69ce89f0 go fmt 2018-09-21 01:56:43 -07:00
Chris Lu 788acdf527 add WIP filer.replicate 2018-09-17 00:27:56 -07:00
Chris Lu 70e67b7c6d add notification log 2018-09-16 11:20:36 -07:00
Chris Lu d923ba2206 renaming msgqueue to notification 2018-09-16 01:18:30 -07:00
Chris Lu 296acc4a0a go fmt 2018-09-09 16:26:11 -07:00
Chris Lu 164091c269 add s3 multipart upload 2018-09-09 16:25:43 -07:00
Chris Lu b23883b3b6 fix a bug on error reporting 2018-08-22 18:19:26 -07:00
Chris Lu f22d6b87e1 go fmt 2018-08-19 15:42:50 -07:00
Chris Lu 748870b03f add "weed scaffold" command 2018-08-19 15:36:30 -07:00
Chris Lu 95b997fb35 go fmt 2018-08-19 15:18:37 -07:00
Chris Lu f827ada811 merge notification config with filer.toml 2018-08-19 15:17:55 -07:00
Chris Lu ee920d4f83 kafka set the right topic 2018-08-19 01:27:30 -07:00
Chris Lu 708acee502 add support for redis cluster 
fix https://github.com/chrislusf/seaweedfs/issues/705
 2018-08-15 00:01:38 -07:00
Chris Lu ae9b81c18f fix tests 2018-08-13 01:33:21 -07:00
Chris Lu f036ef8a3c add filer notification 2018-08-13 01:20:49 -07:00
Chris Lu c81f1cda47 correctly recursively delete folders 2018-08-01 01:26:41 -07:00
Chris Lu d3205a0070 go fmt 2018-07-28 21:02:56 -07:00
Chris Lu 888eb2abb5 filer read write all via locations from MasterClient 2018-07-28 14:51:36 -07:00
Chris Lu 1d779389cb MasterClient replicates all vid locations 2018-07-28 14:22:46 -07:00
Chris Lu 01bcc89803 refactor into MasterClient 2018-07-28 02:10:32 -07:00
Chris Lu a12c7b86b0 broadcast messages of new and deleted volumes 2018-07-27 23:09:55 -07:00
Chris Lu 6f30a78a6c update also delete old chunks if different from latest chunks 2018-07-21 17:40:00 -07:00
Chris Lu 7e2031b18f go fmt 2018-07-21 17:39:10 -07:00
Chris Lu 80d80daf64 set filer2.ErrNotFound for not found entry 2018-07-20 02:14:18 -07:00
Chris Lu d429b6296b fix test 2018-07-19 01:27:45 -07:00
Chris Lu c7228fefa7 add bucket creation and deletion 
1. option for "weed s3 -filer.dir.buckets" to choose a folder for buckets
2. create a bucket
3. delete a bucket, recursively delete all metadata on filer
 2018-07-19 01:21:44 -07:00
Chris Lu 9e974f6a1b
Merge pull request #686 from qszxnp/filer2_dir_cache_bugfix 
bugfix:delete Directory cache when filer2 delete Directory
 2018-07-12 01:17:32 -07:00
聂鹏 1ed41f8b21 bugfix:check Directory attr before create it 2018-07-12 00:53:31 -04:00
聂鹏 c523bc4f07 bugfix:delete Directory cache when filer2 delete Directory 2018-07-11 21:27:25 -04:00
Chris Lu 77fc8c5914 keep alive for gRpc calls 2018-07-03 19:07:55 -07:00
Chris Lu 14c6f12b0c adjust interface parameter name and doc 2018-06-17 13:24:57 -07:00
Chris Lu db0c25b3d8 delete old entry only after successful insert 2018-06-17 13:02:22 -07:00
Chris Lu eb2acd11c2 decouple from viper for filer store 2018-06-17 13:01:57 -07:00
Chris Lu 98110c1697 add collection, replication, ttl to each file 2018-06-10 16:57:32 -07:00
Chris Lu 5c25d29272 support renaming files 2018-06-06 22:11:01 -07:00
Chris Lu 43e3f5724c use fixed list of masters in both filer and volume servers 2018-06-01 00:39:39 -07:00
Chris Lu 0301504184 add mime, use simple insert and update filer store API 
1. add mime type to file in filer
2. purge old chunks if overwrite during insert
 2018-05-30 20:24:57 -07:00
Chris Lu 74332e1a61 minor 2018-05-28 14:32:16 -07:00
Chris Lu 07e0d13d2d filer support reading multiple chunks, with range support 2018-05-28 05:39:12 -07:00
Chris Lu 8db9319a06 refactoring: go fmt, reorg 2018-05-27 23:59:49 -07:00
Chris Lu 8b0718ac92 go vet 2018-05-27 11:56:49 -07:00
Chris Lu 458ada173e go fmt 2018-05-27 11:52:26 -07:00
Chris Lu 03d76479a5 add reds 2018-05-27 11:14:29 -07:00
Chris Lu a6a6b87f1b adjust readme 2018-05-27 11:14:12 -07:00
Chris Lu ab86d263c9 add Cassandra 2018-05-27 00:01:15 -07:00
Chris Lu 603de2d5db skip deletion if entry not found 2018-05-27 00:00:56 -07:00
Chris Lu f124ebab1d fix ordering 2018-05-27 00:00:11 -07:00
Chris Lu 869161a261 support both mysql and postgres 2018-05-26 22:02:49 -07:00
Chris Lu 87b3b84471 simplifying dirhash to 64bit integer 2018-05-26 21:24:03 -07:00
Chris Lu 955eae3500 this works great and support long file names 2018-05-26 14:08:55 -07:00
Chris Lu 2da84ed331 working fine now 
but index length can be improved
 2018-05-26 13:35:56 -07:00
Chris Lu 68bcaff14d mysql can compile, not yet tested! 2018-05-26 05:32:15 -07:00
Chris Lu c1e353491a refactoring variable names 2018-05-26 04:50:55 -07:00
Chris Lu 828e4a5ace fix test error 2018-05-26 04:13:50 -07:00
Chris Lu 9e77563c99 add leveldb store 
1. switch to viper for filer store configuration
2. simplify FindEntry() return values, removing “found”
3. add leveldb store
 2018-05-26 03:49:46 -07:00
Chris Lu c34feca59c refactoring 2018-05-25 23:27:06 -07:00
Chris Lu 72fc001d0d remove unused code 2018-05-25 01:30:16 -07:00
Chris Lu 0a223838bd refactoring 2018-05-25 00:57:25 -07:00
Chris Lu a01557f4cb adjust filer web page 2018-05-24 23:21:12 -07:00
Chris Lu e18c7e160c avoid empty chunk view 2018-05-24 23:19:56 -07:00
Chris Lu d773e11c7a file handler directly read from volume servers 
this mostly works fine now!

next: need to cache files to local disk
 2018-05-24 01:22:37 -07:00
Chris Lu 00d0274fd7 prepare to read from multiple file chunks 2018-05-23 22:28:54 -07:00
Chris Lu e97c60cc10 avoid overwriting variables 2018-05-23 02:36:19 -07:00
Chris Lu 468514f525 reduce logs 2018-05-23 02:36:06 -07:00
Chris Lu 873868cc10 not working now 
need to add file handler
 2018-05-22 04:31:44 -07:00
Chris Lu 7362de9a18 weed mount can work well 
TODO: somehow filer url is returning empty content
 2018-05-22 03:26:38 -07:00
Chris Lu 9dd228747c filer copy added uid/gid 2018-05-21 01:25:30 -07:00
Chris Lu f07482382b able to update file content 
having some issue when vi reports file changed.
 2018-05-21 00:00:28 -07:00
Chris Lu cd47528a75 remove unused code 2018-05-20 17:08:54 -07:00
Chris Lu cc66e25cd2 merge intervals 2018-05-20 17:06:09 -07:00
Chris Lu 793dd81ca2 skip permission checking when creating dir or files 2018-05-19 12:40:24 -07:00
Chris Lu ab4ddb1e0e fix directory creation, directory listing 2018-05-19 12:07:15 -07:00
Chris Lu b303a02461 cp file can work 
1. consolidate to filer_pb.FileChunk
2. dir add file, mkdir
3. file flush, write

updates having issue
 2018-05-16 00:08:44 -07:00
Chris Lu 58954bf46f pass file attributes from filer to mount 2018-05-14 02:02:17 -07:00
Chris Lu c5cf9bd290 properly working filer 2018-05-13 23:56:16 -07:00
Chris Lu f01d5616b3 add better listing directory entries 2018-05-13 14:02:29 -07:00
Chris Lu 75a2b05aa3 remove Size from Attr 2018-05-13 00:11:26 -07:00
Chris Lu fc7f81f6e7 remove nlink property from directory node 2018-05-12 22:42:28 -07:00
Chris Lu 9ecbf92569 more tests 2018-05-12 13:50:11 -07:00
Chris Lu 81af1bafba file store adds memdb implementation 2018-05-12 13:45:29 -07:00
Chris Lu a808df5019 visit parent folder first 2018-05-11 02:27:57 -07:00
Chris Lu 8a1d640dc4 starting filer2 2018-05-11 02:20:15 -07:00